Codecov Report

Attention: 402 lines in your changes are missing coverage. Please review.

Comparison is base (f6bf6b3) 83.48% compared to head (fe72800) 82.31%.

@@            Coverage Diff             @@
##             main    #1219      +/-   ##
==========================================
- Coverage   83.48%   82.31%   -1.18%     
==========================================
  Files          27       27              
  Lines       24082    24313     +231     
==========================================
- Hits        20106    20013      -93     
- Misses       3976     4300     +324     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Evidence

./1206.exe csv-timeline -d ../hayabusa-sample-evtx
...
Total event log files: 584
Total file size: 137.1 MB

Scan wizard:

✔ Which set of detection rules would you like to load? · 2. Core+ (2193 rules) ( status: test, stable | level: medium, high, critical )
✔ Include Emerging Threats rules? (183 rules) · no
✔ Include Threat Hunting rules? (6 rules) · no
✔ Include deprecated rules? (161 rules) · no
✔ Include noisy rules? (0 rules) · no
✔ Include unsupported rules? (40 rules) · no
✔ Include sysmon rules? (997 rules) · no

@hitenkoku
I have a question!

5. All event and alert rules (4442 rules) ( status: * | level: informational+ )

I thought that the 4442 rule would match the total number of files (./rules/hayabusa and ./rules/sigma folders), but is there any reason why it doesn't match?

I thought that the 4442 rule would match the total number of files (./rules/hayabusa and ./rules/sigma folders), but is there any reason why it doesn't match?

The way we currently count the rules is by unique rule IDs, not number of .yml files. So two separate process_creation rules are still treated as 1 sigma rule because originally there was only 1 sigma rule.

@YamatoSecurity @hitenkoku
Thank you for comment! I understand the specifications of the current counting method.
In that case, I think the expected result is that the number of rules displayed in the console (4442) will be less than the total number of yml files? 🤔

@YamatoSecurity @fukusuket Thanks for your comment.

I checked excluded and noisy status in output count.

I fixed in 691ee18 and f8cf6e6 .

@fukusuket Thanks for your comment. I checked noisy rule count.
I fixed noisy rule count miss of wizard output in 0d77142.

Could you check it?

@hitenkoku Thank you so much for quick fix! I confirmed #1219 (comment) case fixed.

✔ Include noisy rules? (0 rules) · no

The number of noisy rules is 0. Is this the expected behavior? Could you check it out?

./hayabusa csv-timeline -d ../hayabusa-sample-evtx/ -o out.csv -C

Scan wizard:

? Which set of detection rules would you like to load? ›
  1. Core (1421 rules) ( status: test, stable | level: high, critical )
  2. Core+ (2299 rules) ( status: test, stable | level: medium, high, critical )
  3. Core++ (3829 rules) ( status: experimental, test, stable | level: medium, high, critical )
  4. All alert rules (4079 rules) ( status: * | level: low+ )
❯ 5. All event and alert rules (4168 rules) ( status: * | level: informational+ )

Scan wizard:

✔ Which set of detection rules would you like to load? · 5. All event and alert rules (4168 rules) ( status: * | level: informational+ )
✔ Include deprecated rules? (186 rules) · no
✔ Include noisy rules? (0 rules) · no
✔ Include unsupported rules? (45 rules) · no
✔ Include sysmon rules? (2050 rules) · yes

Loading detection rules. Please wait.

Excluded rules: 31
Noisy rules: 12 (Disabled)

Deprecated rules: 186 (7.18%) (Disabled)
Experimental rules: 950 (36.67%)
Stable rules: 198 (7.64%)
Test rules: 1443 (55.69%)
Unsupported rules: 45 (1.74%) (Disabled)

Hayabusa rules: 161
Sigma rules: 2430
Total enabled detection rules: 2591

@hitenkoku
Sorry to add some more things. Can I ask the following:

1. When 1-3 (Core, Core+, Core++) is selected, we should not ask the user if they want to include deprecated rules or unsupported rules as they have a different status and cannot be enabled. Can you change it so we only ask the user if they want to enabled depracted and unsupported when 4 or 5 is selected?

2. The rule count should include noisy rules but ignore excluded rules.

3. This one may be hard to implement so if it is too complex then please ignore it. If possible, when asking the user if they want to include certain rules, we should create filters based on the first question. For example, if the user chooses 1. Core ( status: test, stable | level: high, critical ), then when we ask if they want to include noisy rules, we should only count the number of rules that are 1. marked as noisy and 2. has a status of test or stable and a level of high or above.

4. If the number of rules is 0, then we don't need to ask the user anything so should just ignore that question.

5. The final count of rules should be the number of .yml files instead of unique rule IDs. I'm very sorry for this one as we changed over to unique rule IDs but as we are now counting the rules based on number of files I think we should switch back so it makes more sense to the user.

This part:

Loading detection rules. Please wait.

Excluded rules: 1878
Noisy rules: 12 (Disabled)

Stable rules: 69 (7.83%)
Test rules: 812 (92.17%)

Hayabusa rules: 19
Sigma rules: 862
Total enabled detection rules: 881

@YamatoSecurity Thanks for your comment.

I fixed follow comment in c9ad9a0 .

I will implement to other comment.

5. The final count of rules should be the number of .yml files instead of unique rule IDs. I'm very sorry for this one as we changed over to unique rule IDs but as we are now counting the rules based on number of files I think we should switch back so it makes more sense to the user.

This part:

Loading detection rules. Please wait.

Excluded rules: 1878
Noisy rules: 12 (Disabled)

Stable rules: 69 (7.83%)
Test rules: 812 (92.17%)

Hayabusa rules: 19
Sigma rules: 862
Total enabled detection rules: 881

I already implemented following comment. Could you check it?

3. This one may be hard to implement so if it is too complex then please ignore it. If possible, when asking the user if they want to include certain rules, we should create filters based on the first question. For example, if the user chooses 1. Core ( status: test, stable | level: high, critical ), then when we ask if they want to include noisy rules, we should only count the number of rules that are 1. marked as noisy and 2. has a status of test or stable and a level of high or above.

Scan wizard:

✔ Which set of detection rules would you like to load? · 1. Core (1377 rules) ( status: test, stable | level: high, critical )
✔ Include Emerging Threats rules? (175 rules) · no
✔ Include Threat Hunting rules? (0 rules) · no
✔ Include deprecated rules? (105 rules) · no
✔ Include noisy rules? (0 rules) · no
✔ Include unsupported rules? (20 rules) · no
✔ Include sysmon rules? (626 rules) · no

Scan wizard:

✔ Which set of detection rules would you like to load? · 2. Core+ (2193 rules) ( status: test, stable | level: medium, high, critical )
✔ Include Emerging Threats rules? (183 rules) · no
✔ Include Threat Hunting rules? (6 rules) · no
✔ Include deprecated rules? (161 rules) · no
✔ Include noisy rules? (0 rules) · no
✔ Include unsupported rules? (40 rules) · no
✔ Include sysmon rules? (997 rules) · no

@YamatoSecurity I fixed following comment. Could you check it?

1. When 1-3 (Core, Core+, Core++) is selected, we should not ask the user if they want to include deprecated rules or unsupported rules as they have a different status and cannot be enabled. Can you change it so we only ask the user if they want to enabled depracted and unsupported when 4 or 5 is selected?

I fixed in e65e1d4

2. The rule count should include noisy rules but ignore excluded rules.

I fixed in a34f3ae

4. If the number of rules is 0, then we don't need to ask the user anything so should just ignore that question.

I fixed in bdb4b00.

@hitenkoku Thank you so much!

For this:

✔ Which set of detection rules would you like to load? · 1. Core (1423 rules) ( status: test, stable | level: high, critical )
✔ Include Emerging Threats rules? (190 rules) · yes
✔ Include sysmon rules? (642 rules) · yes

Loading detection rules. Please wait.

Excluded rules: 2870
Noisy rules: 12 (Disabled)

Stable rules: 103 (7.75%)
Test rules: 1226 (92.25%)

Hayabusa rules: 19
Sigma rules: 1310
Total enabled detection rules: 1329

If the user chooses 1 and enables both ET rules and sysmon rules, then shouldn't the Total enabled detection rules be the same as Core (1423 rules) ( status: test, stable | level: high, critical ) ?
Why does the rule count decrease from 1423 to 1329?

thanks for your comment.
I will check it.

@hitenkoku Thank you so much!

For this:

✔ Which set of detection rules would you like to load? · 1. Core (1423 rules) ( status: test, stable | level: high, critical )
✔ Include Emerging Threats rules? (190 rules) · yes
✔ Include sysmon rules? (642 rules) · yes

Loading detection rules. Please wait.

Excluded rules: 2870
Noisy rules: 12 (Disabled)

Stable rules: 103 (7.75%)
Test rules: 1226 (92.25%)

Hayabusa rules: 19
Sigma rules: 1310
Total enabled detection rules: 1329

If the user chooses 1 and enables both ET rules and sysmon rules, then shouldn't the Total enabled detection rules be the same as Core (1423 rules) ( status: test, stable | level: high, critical ) ?
Why does the rule count decrease from 1423 to 1329?

@YamatoSecurity Sorry for my late comment.

I checked following problm and fixed in 7aa6ba5.

Could you check it?

For this:

✔ Which set of detection rules would you like to load? · 1. Core (1423 rules) ( status: test, stable | level: high, critical )
✔ Include Emerging Threats rules? (190 rules) · yes
✔ Include sysmon rules? (642 rules) · yes

Loading detection rules. Please wait.

Excluded rules: 2870
Noisy rules: 12 (Disabled)

Stable rules: 103 (7.75%)
Test rules: 1226 (92.25%)

Hayabusa rules: 19
Sigma rules: 1310
Total enabled detection rules: 1329

If the user chooses 1 and enables both ET rules and sysmon rules, then shouldn't the Total enabled detection rules be the same as Core (1423 rules) ( status: test, stable | level: high, critical ) ? Why does the rule count decrease from 1423 to 1329?

@hitenkoku Thank you so much!
1-3 Core rule count is now working as expected.

However, when I choose the following:

✔ Which set of detection rules would you like to load? · 4. All alert rules (4138 rules) ( status: * | level: low+ )
✔ Include deprecated rules? (184 rules) · yes
✔ Include unsupported rules? (45 rules) · yes
✔ Include noisy rules? (1 rules) · yes
✔ Include sysmon rules? (2029 rules) · yes

I get only 4108 rules enabled. Shouldn't it be 4138?

Output:

Loading detection rules. Please wait.

Excluded rules: 133
Noisy rules: 12

Deprecated rules: 184 (4.48%)
Experimental rules: 1551 (37.76%)
Stable rules: 161 (3.92%)
Test rules: 2167 (52.75%)
Unsupported rules: 45 (1.10%)

Hayabusa rules: 84
Sigma rules: 4024
Total enabled detection rules: 4108

Also, a similar thing with 5. All event and alert rules (4241 rules) ( status: * | level: informational+ ):

✔ Which set of detection rules would you like to load? · 5. All event and alert rules (4241 rules) ( status: * | level: informational+ )
✔ Include deprecated rules? (186 rules) · yes
✔ Include unsupported rules? (45 rules) · yes
✔ Include noisy rules? (12 rules) · yes
✔ Include sysmon rules? (2054 rules) · yes

Loading detection rules. Please wait.

Excluded rules: 31
Noisy rules: 12

Deprecated rules: 186 (4.42%)
Experimental rules: 1553 (36.89%)
Stable rules: 246 (5.84%)
Test rules: 2180 (51.78%)
Unsupported rules: 45 (1.07%)

Hayabusa rules: 173
Sigma rules: 4037
Total enabled detection rules: 4210

Total enabled detection rules should be 4241, not 4210, correct?

@YamatoSecurity I'm sorry to bother you again.

I fixed follow problem in fe72800.

I also confirmed 1.-5. wizard count is matched with rule count output.

Could you check it?

However, when I choose the following:

✔ Which set of detection rules would you like to load? · 4. All alert rules (4138 rules) ( status: * | level: low+ )
✔ Include deprecated rules? (184 rules) · yes
✔ Include unsupported rules? (45 rules) · yes
✔ Include noisy rules? (1 rules) · yes
✔ Include sysmon rules? (2029 rules) · yes

I get only 4108 rules enabled. Shouldn't it be 4138?

Output:

Loading detection rules. Please wait.

Excluded rules: 133
Noisy rules: 12

Deprecated rules: 184 (4.48%)
Experimental rules: 1551 (37.76%)
Stable rules: 161 (3.92%)
Test rules: 2167 (52.75%)
Unsupported rules: 45 (1.10%)

Hayabusa rules: 84
Sigma rules: 4024
Total enabled detection rules: 4108

Also, a similar thing with 5. All event and alert rules (4241 rules) ( status: * | level: informational+ ):

✔ Which set of detection rules would you like to load? · 5. All event and alert rules (4241 rules) ( status: * | level: informational+ )
✔ Include deprecated rules? (186 rules) · yes
✔ Include unsupported rules? (45 rules) · yes
✔ Include noisy rules? (12 rules) · yes
✔ Include sysmon rules? (2054 rules) · yes

Loading detection rules. Please wait.

Excluded rules: 31
Noisy rules: 12

Deprecated rules: 186 (4.42%)
Experimental rules: 1553 (36.89%)
Stable rules: 246 (5.84%)
Test rules: 2180 (51.78%)
Unsupported rules: 45 (1.07%)

Hayabusa rules: 173
Sigma rules: 4037
Total enabled detection rules: 4210

Total enabled detection rules should be 4241, not 4210, correct?

@fukusuket @YamatoSecurity thanks for your review.
I will merge it.